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Figure 1: Nodes and physical connectivity 
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Figure 2a) JOIN Protocol: PROCLAIM message 
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Figure 2b) JOIN Protocol; JOIN message 
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Figure 2d) JOIN Protocol: PTC.ACK message 
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Figure 2e) JOIN Protocol: COMMTT.BCAST message ^ 
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Figure 2f) JOIN Protocol: COMMIT and COMMIT_BCAST_ACK messages 
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Figure 2g) JOIN Protocol: new group formed after completion of protocol 
Figure 2: JOIN protocol 
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Figure 3a) DEATH Protocol: initial state: heartbeat ring 
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Figure 3b) DEATH Protocol: DEATH message 




Hgure 3c) DEATH Protocol: PTC message 
Figure 3: DEATH protocol 




NODE.CONNECnVTTY 
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Figure 4b) Node Reachability Protocol: GROUP_CONNECnVITY message 
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Figure 4c) Node Reachability Protocol: forwarding of GROUP.CONNECnvnY message 
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5a) Initial situation 
Figure 5: Topology Propagation Scenario: node death 
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5b) Node 2 dies: Nodes 1, 3, and 4 fonn AMG KJ. 
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5c) GCM for AMG A_2 is propagated to all nodes 
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Figured) Inconsistency caused by quick daemon restart in the presence of different detection times 
tor each network: the daemon on node 1 goes down and is restarted, but this is never detected 
by node 2. 
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Figure f) Inconsistency caused by temporary communication problem in an adapter: node 1 
IS not the GL or CP m its group. Node 1 never notices other nodes as down, though the others 
do see node Ibemg unreachable. 
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Figure ^) Inconsistency caused by temporary communication problem in an adapter: node 1 
is the GL or CP in its group. Node 1 never sees the other nodes as unreachable, while the 
others do see node 1 as unreachable for a period. 
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Figure "#) Adapter IDs aiid Group IDs. An adapter ID has the IP address 
of the adapter and and mstance number. The Group ID has the IP address 
of the Group Leader and an instance number that changes each time the 
group changes. 
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Figure #) Format of the protocol packets that are sent over the network 
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Figure ^) Adapter and Group IDs when the daemon at node 1 terminates and is 
restarted* 
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Figure A "live" node detects that a remote daemon restarted.The Group ID of the 
message is different fix)m node 2's, while the address of the sender is listed on node 2's 
group membership. 
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Figure Ma) A daemon that is restarted detects that a previous instance used to belong to 
an AMG because of heartbeat messages that it receives while in a singleton group. 
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Figure Sfo) Continuation 
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Figure ^a) Solution to the Quick Communication Interruption Problem, initial state: 
nodes 1,2, and3 are part of the same AMG,. Node 3's adapter suffers a temporary 
failure. ^ ^ 
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Figure'tfb) Solution to the Quick Communication Interruption Problem. Node 3's 
adapter suffers a temporary failure. Node 2 commits a new AMG, while node 3 is still 
in the process of missing HBs from its neighbor 
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Figure fee) Solution to the Quick Communication Interruption Problem. Node 3 sends 
a PTC when it stops receiving HBs from its upstream nei^bor. The PTCs are rejected 
because of the discrepancy in the last_stable_group results. 
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Figore lid) Solution to the Quick Communication Interruption Problem. Since node 3 
does not get repUes to its PTC messages, it is forced to form a singleton group. At this 
pomt, it updates last_stable_group. From then on node 3's PTC are accepted again. 
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Figure «a) Solution to the Quick Communication Interruption Problem, initial state: 
nodes 1,2, and3 are part of the same AMG,. Node Ts adapter suffers a tempoiary 
failure. 
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Figure 5lb) Solution to the Quick Communication Interruption Problem. Node I's 
adapter suffers a temporary failure. Node 3 commits a new AMG, while node 1 is still 
in the process of missing HBs from its neighbor 
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Figure ffe:) Solution to the Quick Communication Interruption Problem. Node 1 
dissolves its group and forms a singleton unstable group. Note that because the 
group is unstable, there is no change in last_stable_group. 
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Figure ifild) Solution to the Quick Communication Interruption Problem. Node 3 sends 
a PTC when node 1 responds the PROCLAIM message with a JOIN. The PTCs are rejected 
because of the discrepancy in the last_stable_group results. 
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Figure We) Solution to the Quick Communication Interruption Problem, Since node 3 
does not get replies to its PTC messages, it is eventually forced to form a singleton groups 
At this point, it updates last_stable_^oup. From then on node 3's PTC are accepted again. 



